<html>
<head><meta charset="utf-8"><title>test marks flakey? · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html">test marks flakey?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="230346699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230346699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Mcguigan <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230346699">(Mar 15 2021 at 13:49)</a>:</h4>
<p>Are test marks known to ever be flakey? I'm pretty sure I have seen false failures (at least once) for mark not hit when running the tests locally, and I just saw it in CI - <a href="https://github.com/rust-analyzer/rust-analyzer/pull/8027">https://github.com/rust-analyzer/rust-analyzer/pull/8027</a>. Re-running caused the CI tests to pass. Specifically it is the <code>expected_type_if_let_without_leading_char</code> mark. I don't think there is anything in that test that is non-deterministic.</p>



<a name="230347138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347138">(Mar 15 2021 at 13:51)</a>:</h4>
<p>Not AFAIK</p>



<a name="230347208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347208">(Mar 15 2021 at 13:52)</a>:</h4>
<p>Marks might falsely pass, but they should never falsely fail</p>



<a name="230347391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347391">(Mar 15 2021 at 13:53)</a>:</h4>
<p>That being said, we've recently switched from an in-tree impl to one from <a href="http://crates.io">crates.io</a>. they should be equvalent, but maybe they arent'</p>



<a name="230347577"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347577" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347577">(Mar 15 2021 at 13:54)</a>:</h4>
<p><code>cargo test --package ide_completion --lib -- context::tests::expected_type_if_let_without_leading_char --exact --nocapture</code> fails for me locally reliably</p>



<a name="230347620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347620">(Mar 15 2021 at 13:54)</a>:</h4>
<p>so yeah, you just got lucky on CI :)</p>
<p>(</p>



<a name="230347657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230347657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230347657">(Mar 15 2021 at 13:55)</a>:</h4>
<p>/me thinks if we should enable precise thread-local marks?</p>



<a name="230348300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230348300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230348300">(Mar 15 2021 at 13:59)</a>:</h4>
<p>Yeah, let's try to activate <a href="https://github.com/matklad/cov-mark/blob/master/Cargo.toml#L20">this feature</a> in all test-mark using crates. If it works, then let's roll with it.</p>



<a name="230349298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230349298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230349298">(Mar 15 2021 at 14:04)</a>:</h4>
<p>Right, what do we do with that mark?</p>



<a name="230350922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230350922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Mcguigan <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230350922">(Mar 15 2021 at 14:14)</a>:</h4>
<p>So it does fail locally  for me too when running that test in isolation. But <code>cargo test -p ide_completion</code> is flakey, mostly passing (although I have seen one failure in a handful of runs).</p>



<a name="230351653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230351653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230351653">(Mar 15 2021 at 14:19)</a>:</h4>
<p>Is that even with <a href="https://github.com/rust-analyzer/rust-analyzer/pull/8029">https://github.com/rust-analyzer/rust-analyzer/pull/8029</a>?</p>



<a name="230353276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230353276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Mcguigan <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230353276">(Mar 15 2021 at 14:29)</a>:</h4>
<p>No that is master. I'm looking at your PR now.</p>



<a name="230355406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230355406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Mcguigan <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230355406">(Mar 15 2021 at 14:41)</a>:</h4>
<p>I've confirmed it consistently fails on my machine with the <code>thread-local</code> feature. Thanks for the quick help resolving this!</p>



<a name="230356992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230356992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230356992">(Mar 15 2021 at 14:51)</a>:</h4>
<p><span class="user-mention" data-user-id="275435">@Josh Mcguigan</span> so the way mark work is that each mark is an atomic counter, incremented by each <code>hit</code>. The check then looks at the marks at the start and end of the scope, and makes sure that there was at least one increment. So, if the mark is incremented by an unrelated test, that would get counted as a hit. </p>
<p>This is architectured to allow spawning threads from within the test, and get them pass. There's a thread local opt-out. If all our tests actually work with thread-local, I am tempted to release a 2.0 version, which makes thread-local based impl default, and uses a feature to opt-into non-thread-safetyz</p>



<a name="230357938"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230357938" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Mcguigan <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230357938">(Mar 15 2021 at 14:56)</a>:</h4>
<p>Good to know! Just to make sure I am understanding, does this mean that when we enable <code>thread-local</code>, if we had a test that spawned a thread and that thread hit the mark we'd get a false-fail?</p>



<a name="230358014"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/test%20marks%20flakey%3F/near/230358014" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/test.20marks.20flakey.3F.html#230358014">(Mar 15 2021 at 14:56)</a>:</h4>
<p>right</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>